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An  Analysis  of  SEI  Software  Process 
Assessment  Results:  1987-1991 


Abstract:  This  report  focuses  on  the  results  of  SEI  software  process  assessments 
conducted  over  a  four-year  period  beginning  in  1987.  It  characterizes  the  software 
processes  used  by  software  managers  and  practitioners  at  the  assessed  sites  and 
classifies  issues  identified  during  the  assessments.  The  basis  for  the  characterization  and 
classification  is  a  software  process  maturity  model  developed  by  the  SEI.  This  report 
contributes  to  the  existing  body  of  knowledge  on  the  state  of  practice  of  software 
engineering  in  the  United  States  by  characterizing  the  sites  from  a  software  process 
maturity  perspective  and  profiling  site  software  process  weaknesses.  The  data  analyzed 
are  drawn  from  SEI  software  process  assessments  of  59  government  and  industry 
software  sites.  This  work  is  an  analysis  of  existing  assessment  data  rather  than  a 
designed  study.  The  participating  sites  were  not  randomly  selected;  accordingly,  they  do 
not  necessarily  constitute  a  statistically  valid  sampling  of  the  U.S.  software  industry. 

1  Overview 

1.1  Scope  and  Objectives 

This  report  focuses  on  the  state  of  practice  of  software  engineering  from  a  software  process 
perspective.  It  characterizes  the  software  processes  used  by  software  managers  and 
practitioners  using  a  five-level  process  maturity  model  developed  at  the  SEI,  and  classifies 
process  issues  identified  during  SEI  assessments  conducted  at  59  government  and  industry 
software  sites.  The  scope  of  this  report  does  not  explicitly  consider  other  important 
determinants  of  software  supplier  performance  such  as  human  resource  management, 
automation,  business  strategy  and  practices. 

A  commitment  to  improve  key  software  processes  on  a  continuing  basis  is  rapidly  becoming 
a  high  priority  for  many  U.S.  software  organizations.  Among  the  reasons  for  this  are: 

1.  Process  capability  is  being  increasingly  recognized  (across  many  industries)  as  a 
key  determinant  of  performance  and  a  source  of  competitive  advantage, 

2.  Software  suppliers  (both  government  and  industry)  are  subject  to  intensifying 
competitive  pressures,  and 

3.  Software  purchasers  are  becoming  increasingly  sophisticated  and  demanding. 

Of  particular  importance  to  the  U.S.  Department  of  Defense  (DoD)  software  community  is 
the  increasing  use  by  the  DoD  of  SEI-developed  methods  such  as  software  capability 
evaluation  for  identifying  capable  contractors  during  the  acquisition  phase  and  for  monitoring 
the  results  of  process  improvement  programs  during  contract  performance. 

The  objective  of  this  report  is  to  provide  a  baseline  characterization  of  the  state  of  software 
process  maturity  for  a  group  of  59  U.S.  government  and  industry  software  sites.  A  clear 
understanding  of  current  software  process  strengths  and  weaknesses  is  an  important  initial 
step  towards  formulating  plans  to  improve  them  in  an  orderly,  progressive  and  sustainable 
way. 
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1.2  Basis  for  Analysis  and  Relation  to  Previous  Work 


This  report  is  based  on  data  obtained  from  59  SEI  assessments  conducted  over  a  50  month 
period  beginning  in  1987.  Collectively,  these  assessments  closely  examined  296  software 
projects  and  involved  approximately  2,500  software  managers  and  practitioners  in 
discussions  of  software  process-related  issues  in  their  organizations.  Just  over  half  of  the 
sites  assessed  are  industrial  organizations  working  under  contract  to  the  DoD;  the 
remainder  are  commercial  and  government  software  organizations. 

This  work  constitutes  an  analysis  and  review  of  available  data  (resulting  from  the  conduct  of 
SEI  assessments),  as  opposed  to  a  designed  study.  One  of  the  key  characteristics  of  a 
designed  study  is  the  advance  identification  of  a  set  of  hypotheses  to  be  tested,  along  with 
criteria  by  which  the  appropriate  tests  can  be  constructed  and  carried  out.  In  contrast, 
assessments  primarily  assist  organizations  in  identifying  key  process-related  improvement 
needs  and  taking  appropriate  corrective  actions.  Consequently,  while  we  believe  that  the 
information  presented  is  significant,  there  are  limits  to  what  we  can  expect  to  learn  or  infer 
from  this  data. 

In  conducting  the  analysis  and  formulating  our  conclusions,  information  of  three  types 
was  considered: 

•  Project  manager  responses  to  the  maturity  questionnaire  [Humphrey  87]  and  a 
demographic  data  collection  form. 

•  Assessment  findings  and  maturity  level  ratings  determined  by  the  teams 
conducting  the  assessments.  The  participating  sites  provided  the  SEI  with  either 
the  full  final  assessment  report  or  the  findings  briefing. 

•  The  collective  knowledge  and  experience  that  the  SEI  has  acquired  as  a  result  of 
its  involvement  in  the  development  and  application  of  the  software  process  maturity 
model. 

The  analysis  consisted  of  characterizing  the  projects  and  sites  through  selected 
demographics,  software  process  maturity  profiles,  and  frequency  distributions  of  key 
process  deficiencies  identified  by  the  assessment  teams. 

This  report  updates  and  refines  an  earlier  SEI  review  of  assessment  data 
[Humphrey  89b].  The  most  significant  ways  in  which  this  report  differs  from  its  predecessor 
are: 


•  This  review  is  based  on  a  larger  set  of  SEI  assessment  results  (59  sites  versus  10 
sites  and  296  projects  versus  55  projects). 

•  The  use  of  selected  demographics  to  characterize  the  participating  sites. 

•  The  use  of  a  site-based  software  process  maturity  profile  (previous  maturity 
profiles  have  been  project-based). 

•  The  inclusion  of  an  analysis  of  assessment  findings  which  shows  the  frequency 
with  which  key  process  deficiencies  were  identified  by  assessment  teams. 
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Because  this  report  is  based  on  an  analysis  of  data  which  includes  most  of  the  SEI-assisted 
assessment  data  considered  in  Humphrey,  Kitson,  and  Kasse’s  The  State  of  Software 
Engineering  Practice  [Humphrey  89b],  it  would  be  inappropriate  to  view  the  two  reports  as 
describing  distinct  snapshots  of  the  state  of  practice  in  1989  and  1991  respectively.  Rather, 
we  view  this  report  as  a  refinement  and  extension  of  Humphrey,  Kitson,  and  Kasse’s  report, 
based  on  a  larger  set  of  assessment  results  and  the  introduction  of  additional  analyses. 

This  report  is  organized  into  4  sections.  Sections  1  and  2  provide  background  and  context 
for  the  analysis  of  assessment  results.  Section  3  describes  the  data  collected,  characterizes 
the  projects  and  sites  which  participated,  and  describes  the  software  process  maturity  status 
of  the  participating  sites.  Section  4  presents  an  analysis  of  key  process  weaknesses 
identified  by  the  assessments. 

1.3  Summary  of  Main  Points 

The  analysis  of  data  collected  from  the  participating  sites  shows  that  most  of  the  software 
work  being  performed  is  conducted  at  the  initial  level  of  process  maturity.  While  it  is 
undoubtedly  true  that  some  good  results  are  being  achieved,  it  is  also  true  that  continuing  to 
operate  at  low  levels  of  process  maturity  (levels  1  and  2)  is  a  risk  for  software  supplier 
organizations  in  the  face  of  an  increasingly  competitive  industry,  increasingly  sophisticated 
and  demanding  customers,  and  rising  complexity  of  systems  being  attempted.  Similarly, 
these  results  suggest  improvement  opportunities  for  organizations  searching  for  ways  to 
become  increasingly  effective  and  efficient. 

On  average,  site  assessments  identified  7.6  findings  and  9.3  KPA  findings.  Because  of  this, 
site  findings  usually  span  at  least  two  maturity  levels. 

The  five  most  frequently  occurring  findings  areas  are  product  engineering,  project  planning, 
organization  process  definition,  project  tracking  and  oversight,  and  training  program.1 

The  five  least  frequently  occurring  findings  areas  are  process  change  management,  defect 
prevention,  subcontract  management,  quality  management,  and  peer  reviews. 


■•The  findings  areas  used  are  drawn  from  the  SEI's  Capability  Maturity  Model  for  Software  VI  .0.  See 
Appendix  B  for  additional  details. 
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2  Background  and  Context 
2.1  SEI  Focus  on  Process 


The  Software  Engineering  Institute  (SEI)  was  established  by  the  DoD  in  1984  to  provide 
leadership  in  advancing  the  state  of  the  practice  of  software  engineering  to  improve  the 
quality  of  systems  that  depend  on  software.  Since  early  1987  the  SEI  has  focused  on 
software  process  as  a  principle  means  of  accelerating  the  maturity  of  software  engineering 
as  a  practice  and  facilitating  the  effective  introduction  of  available  technology.  This  focus  is 
based  on  the  premises  that  the  process  of  producing  and  evolving  software  products  can  be 
defined,  managed,  measured,  and  progressively  improved,  and  that  the  quality  of  a  software 
product  is  largely  governed  by  the  quality  of  the  process  used  to  create  and  evolve  it. 

The  SEI  approach  has  emphasized  the  following: 

1 .  Developing  and  evolving  a  software  process  maturity  model, 

2.  Developing  and  transitioning  an  evaluation  method  for  DoD  software  acquisition 
agencies  and  their  prime  contractors, 

3.  Developing  and  transitioning  a  companion  assessment  method  for  use  by  software 
suppliers  in  assessing  their  software  engineering  capabilities  and  determine 
improvement  needs  and  priorities,  and 

4.  Periodically  publishing  reports  summarizing  the  results  of  SEI  assessments. 

2.2  Software  Process  Management  Overview 

The  software  process  is  the  set  of  activities,  methods,  and  practices  that  guide  people  in  the 
production  of  software.  An  effective  process  must  consider  the  relationships  of  the  required 
tasks,  the  tools  and  methods,  and  the  developers’  skills,  training,  and  motivation. 

Software  process  management  is  the  application  of  process  engineering  concepts, 
techniques,  and  practices  to  explicitly  monitor,  control,  and  improve  the  software  process.  It 
is  only  one  of  several  activities  that  must  be  effectively  performed  for  software-producing 
organizations  to  be  consistently  successful.  Capable  and  motivated  technical  people  are 
critical;  knowledge  of  the  ultimate  application  environment  is  needed,  as  is  detailed 
understanding  of  the  end  user’s  needs  [Curtis  88].  Even  with  all  these  capabilities,  however, 
inattention  to  software  management  problems  will  likely  result  in  disappointing  organizational 
performance.  A  more  comprehensive  discussion  of  the  role  and  significance  of  software 
process,  the  discipline  of  software  process  management,  and  software  process 
improvement  methods  is  provided  in  Humphrey’s  Managing  the  Software  Process 
[Humphrey  89a]  and  Kitson  and  Humphrey’s  The  Role  of  Assessment  in  Software  Process 
Improvement  [Kitson  89]. 

This  view  of  process  and  process  management  has  led  to  the  development  of  a  process 
maturity  model  (described  in  Section  2.3),  a  related  software  process  maturity  questionnaire 
[Humphrey  87],  and  a  software  process  assessment  method  (described  in  Section  2.4). 
These  form  key  elements  of  SEI’s  software  process  improvement  framework. 
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2.3  SEI  Software  Process  Maturity  Model 


The  extent  to  which  a  software  organization  has  adopted  and  institutionalized  a  continuous 
improvement  focus  can  be  characterized  with  the  aid  of  the  software  process  maturity  model 
shown  in  Figure  2-1.  This  five-level  model  identifies  the  key  improvements  required  at  each 
level  and  establishes  a  priority  order  for  moving  to  higher  levels  of  process  maturity. 


Level 

Characteristic 

Key  Challenges 

Result 

5 

Optimizing 

Improvement  fed 
back  Into  process 

Human  intensive  process 
Maintain  organization  at 
optimizing  level 

Productivity^ 
&  Quality  M 

ill 

4 

Managed 

(Quantitative) 

Measured  process 

Changing  technology 

Problem  analysis 

Problem  prevention 

:  - 

3 

Defined 

(Qualitative) 

Process  defined  and 
Institutionalized 

Process  measurement 

Process  analysis 

Quantitative  quality  plans 

2 

Repeatable 

(Intuitive) 

Process  dependent 
on  individuals 

Training 

Technical  practices 

Process  focus 

J 

JFl 

1 

Initial 

(Ad  hoc  /  chaotic) 

Project  management 

Project  planning 

Configuration  management 
SQA 

f  Risk 

Figure  2-1  SEI  Software  Process  Maturity  Model 


At  the  Initial  level  (level  1),  an  organization  can  be  characterized  as  having  an  ad  hoc,  or 
possibly  chaotic  process.  Typically,  the  organization  operates  without  formalized 
procedures,  cost  estimates,  and  project  plans.  Even  if  formal  project  control  procedures 
exist,  there  are  no  management  mechanisms  to  ensure  that  they  are  followed.  Tools  are  not 
well  integrated  with  the  process,  nor  are  they  uniformly  applied.  In  addition,  change  control 
is  lax,  software  quality  assurance  (if  present  at  all)  is  ineffective,  and  senior  management  is 
not  exposed  to  or  does  not  understand  the  key  software  problems  and  issues.  When 
projects  do  succeed,  it  is  generally  due  to  the  heroic  efforts  of  a  dedicated  team  rather  than 
to  the  process  capabilities  of  the  organization. 

An  organization  at  the  Repeatable  level  (level  2)  has  established  basic  project  controls  such 
as  project  management,  management  oversight,  quality  assurance,  and  change  control.  The 
strength  of  the  organization  stems  from  its  experience  at  doing  similar  work,  but  it  faces 
major  risks  when  presented  with  new  challenges.  The  organization  has  frequent  quality 
problems  and  lacks  an  orderly  framework  for  improvement. 
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At  the  Defined  level  (level  3),  the  organization  has  laid  the  foundation  for  examining  the 
process  and  deciding  how  to  improve  it.  A  group  has  been  established  to  focus  and  lead  the 
process  improvement  efforts,  to  keep  management  informed  on  the  status  of  these  efforts, 
and  to  facilitate  the  introduction  of  a  family  of  software  engineering  methods  and 
technologies.  Such  groups  are  typically  known  as  Software  Engineering  Process  Groups,  or 
SEPGs. 

The  Managed  level  (level  4)  builds  on  the  foundation  established  at  the  defined  level.  When 
the  process  is  defined,  it  can  be  examined  and  improved  but  there  is  little  data  to  indicate 
effectiveness.  Thus,  an  organization  at  this  level  has  established  a  minimum  set  of 
measurements  for  the  quality  and  productivity  parameters  of  each  key  task.  The 
organization  has  also  established  a  process  database  with  resources  to  manage  and 
maintain  it,  to  analyze  the  data,  and  to  advise  project  members  on  its  meaning  and  use. 

Two  requirements  are  fundamental  to  advance  from  the  Managed  to  the  Optimizing  level 
(level  5).  Data  gathering  should  be  automated,  and  management  should  redirect  its  focus 
from  the  product  to  process  analysis  and  improvement.  At  the  optimizing  level,  the 
organization  has  the  means  to  identify  the  weakest  process  elements  and  strengthen  them, 
data  are  available  to  justify  applying  technology  to  various  critical  tasks,  and  numerical 
evidence  is  available  on  the  effectiveness  with  which  the  process  has  been  applied.  The  key 
additional  activity  at  the  optimizing  level  is  rigorous  defect  cause  analysis  and  defect 
prevention. 

These  maturity  levels  have  been  selected  because  they  do  the  following: 

•  Reasonably  represent  the  historical  phases  of  evolutionary  improvement  of  actual 
software  organizations. 

•  Represent  a  measure  of  improvement  that  is  reasonable  to  achieve  from  the  prior 
level. 

•  Suggest  interim  improvement  goals  and  progress  measures. 

•  Make  obvious  a  set  of  immediate  improvement  priorities,  once  an  organization’s 
status  in  this  framework  is  known. 

While  there  are  many  aspects  to  the  transition  from  one  maturity  level  to  another,  the  basic 
objective  is  to  achieve  a  controlled  and  measured  process  as  the  foundation  for  continuous 
improvement. 

It  has  been  our  experience  that  when  software  organizations  are  assessed  against  this 
maturity  framework,  the  assessment  method  enables  reasonably  accurate  placement  of 
them  on  the  maturity  scale  and  helps  to  identify  key  improvement  needs.  In  practice  we  find 
that  when  management  focuses  on  the  few  highest  priority  items,  their  organizations 
generally  make  rapid  improvement  in  being  able  to  produce  quality  software  products  on 
time  and  within  budget.  While  the  use  of  tools  and  technology  can  enhance  software 
engineering  capability,  such  investments  are  generally  of  limited  value  for  organizations  with 
low-maturity  software  processes. 
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Humphrey  and  Kitson  provide  more  comprehensive  descriptions  of  software  process 
management  and  the  maturity  model  ([Humphrey  89a],  [Kitson  89]).  It  should  be  noted  that 
the  SEI  has  published  two  technical  reports  which  provide  an  elaboration  and  refinement  of 
the  maturity  model  discussed  above  ([Paulk  91],  [Weber  91]),  referred  to  as  the  SEI 
Capability  Maturity  Model  for  Software  (CMM). 

2.4  Assessing  Software  Organizations 

There  are  a  number  of  ways  the  software  process  maturity  model  can  be  applied.  The  SEI 
has  developed  and  applied: 

•  SEI-assisted  assessments 

•  self-assessments 

•  assessment  tutorials 

•  SEI-licensed  vendor  assessments 

•  capability  evaluations 

While  all  these  methods  have  contributed  to  the  SEI’s  base  of  knowledge  and  understanding 
of  the  software  process,  the  data  in  this  report  was  obtained  solely  from  SEI-assisted 
assessments  and  self-assessments.1  These  methods  are  described  in  Sections  2.4.1  and 
2.4.2,  respectively.  A  more  comprehensive  discussion  of  the  principles  of  software  process 
management,  the  role  of  assessment  in  software  process  improvement,  and  how 
assessments  are  conducted  can  be  found  in  Kitson  and  Humphrey  [Kitson  89]. 


2.4.1  SEI-Assisted  Assessments 

An  SEI-assisted  assessment  is  an  appraisal  of  a  site’s  current  software  process  by  a  trained 
team  of  experienced  software  professionals.  Typically,  a  team  is  composed  of  four  to  six  SEI 
professionals  and  one  to  three  professionals  from  the  organization  being  assessed.  The 
method  for  conducting  such  assessments  has  been  developed  by  the  SEI  [Olson  89].  The 
assessment  team  receives  SEI  training  prior  to  conducting  the  assessment.  The  goal  is  to 
facilitate  improvement  of  the  organization’s  software  process.  Typically,  four  to  six  projects 
are  examined  during  an  assessment.  The  assessment  team  identifies  the  most  important 
software  process  issues  currently  facing  the  organization  and  formulates  recommendations 
to  deal  with  them. 

SEI-assisted  assessments  are  conducted  in  accordance  with  an  assessment  agreement 
signed  by  the  SEI  and  the  organization  being  assessed.  This  agreement  provides  for  senior 
management  involvement,  organizational  representation  on  the  assessment  team, 
confidentiality  of  results,  and  follow-up  actions. 


1  There  was  limited  vendor  assessment  data  available  at  the  time  the  analysis  for  this  report  was 
performed;  capability  evaluations,  as  defined  by  the  SEI,  do  not  produce  findings  or  a  site  maturity 
rating;  assessment  tutorials  do  not  produce  findings  or  a  site  maturity  rating. 
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The  SEI  has  conducted  SEI-assisted  assessments  since  February  1987  and  uses  the 
information  gained  to  refine  and  improve  the  assessment  method  and  to  add  to  its 
assessment  database. 


2.4.2  Self-Assessments 

Self-assessments  are  SEI  assessments  conducted  with  little  or  no  direct  SEI  involvement. 
Self-assessment  teams  are  composed  primarily  of  software  professionals  from  the 
organization  being  assessed,  with  possibly  one  or  two  SEI  assessment  coaches  present. 
The  context,  objective,  and  degree  of  validation  are  the  same  as  for  SEI-assisted 
assessments.  Organizations  which  have  received  SEI  self-assessment  training  agree  to 
provide  the  SEI  with  the  results  of  their  assessments.  These  results  are  added  to  SEI's 
assessment  database. 


CM  U/S  El-9  2-TR-24 


9 


3  Selected  Demographics  and 
Site  Maturity  Level  Profile 


3.1  Overview  of  Participating  Organizations  and  Sites 

To  minimize  confusion  we  shall  henceforth  use  the  term  site  to  refer  to  that  subset  of  an 
organization  which  was  the  focus  of  the  assessment.  In  general,  a  site  was  one  part  of  a 
larger  corporate  entity.  In  some  cases,  the  organizations  were  very  large  (e.g.,  U.S.  Air 
Force,  Unisys)  and  had  two  or  more  sites  which  contributed  assessment  results  to  the  SEI. 

The  data  for  this  report  is  taken  from  SEI  assessments  of  59  sites  during  the  period 
February  1987  through  March  1991.1  Figure  3-1  shows  a  breakout  of  how  the  sites 
characterized  themselves  according  to  site  type. 


51% 

Figure  3-1  Participating  Site  Types 


I  Other 

□  Federal  Agency 
H  Military  Service 
El  DoD  Contractor 
EH  Commercial 


‘'Demographic  data  was  not  available  for  the  complete  set  of  59  site  assessments  and  296  projects; 
we  received  demographic  data  from  36  sites  covering  170  projects. 
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Appendix  A  lists  the  twenty-seven  participating  organizations.  Nineteen  of  the  twenty 
participating  industrial  organizations  were  ranked  among  the  top  one  hundred  prime  DoD 
contractors  for  fiscal  year  1990  based  on  net  contract  value,  ten  were  among  the  top  twenty, 
and  sixteen  were  among  the  top  fifty  [Carroll  91].  Collectively,  these  nineteen  organizations 
were  awarded  close  to  forty  billion  dollars  worth  of  contracts  in  fiscal  year  1990.  Appendix  A 
also  includes  the  individual  organization  rankings  for  fiscal  year  1990  and  total  contract 
value  awarded. 

Figure  3-2  shows  how  the  site  assessments  considered  in  this  report  are  distributed  over 
time.  In  total,  we  have  data  from  thirteen  SEI-assisted  assessments  and  forty-six  self- 
assessments.  All  assessment  teams  received  training  in  the  SEI  assessment  method 
directly  from  the  SEI.  Some  sites  which  conducted  SEI-assisted  assessments  in  1987  or 
1988  were  re-assessed  (either  by  SEI-assisted  assessment  or  self-assessment);  the  more 
recent  assessment  results  have  been  considered  in  this  work.  Hence,  each  site  is 
represented  by  exactly  one  set  of  assessment  results.  For  this  reason,  the  counts  of  SEI- 
assisted  assessments  conducted  in  1987  and  1988  do  not  reflect  the  actual  number  of  such 
assessments  conducted  during  those  years.  (There  were  actually  four  conducted  in  1987 
and  six  conducted  in  1988.) 


I  SEI-Assisted  Assessments  EH  Self-Assessments 
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Figure  3-2  Time  Distribution  of  Site  Assessments 
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3.2  Statistical  Considerations 


This  work  is  an  analysis  of  existing  assessment  data  rather  than  a  designed  study.  The 
participating  sites  were  not  statistically  selected  and  do  not  therefore  constitute  a  random 
sample.  Accordingly,  we  cannot  claim  that  the  data  presented  or  the  inferences  drawn  in  this 
report  are  representative  of  the  entire  U.S.  software  industry. 

3.3  Selected  Project  Demographics 

To  characterize  the  set  of  projects  selected  for  the  focus  of  each  site  assessment  the 
following  views  of  the  available  demographic  data  have  been  prepared:  the  lifecycle  phase 
of  the  project,  the  product  size  in  thousands  of  source  lines  of  code  (KSLOC),  and  the  peak 
project  staffing.  The  demographic  data  was  provided  by  the  site  project  managers  who 
responded  to  the  maturity  questionnaire  for  their  projects. 

The  first  chart,  Figure  3-3,  shows  the  lifecycle  phase  of  the  project  at  the  time  of  the 
assessment.  Since  an  assessment  focuses  on  how  the  software  work  is  actually  conducted 
(as  opposed  to  how  it  was  planned  or  how  it  should  have  been  done),  current  lifecycle 
phase  was  one  of  the  project  selection  considerations.  In  general,  projects  which  had 
progressed  beyond  the  requirements  analysis  phase  were  considered  more  desirable 
candidates  for  inclusion  in  the  scope  of  the  assessment.  Because  of  the  lifecycle  categories 
used,  we  can  also  differentiate  between  "new  development"  (requirements  analysis  or 
development  and  test  phases)  and  "maintenance"  (production  and  deployment  phase) 
projects.  As  Figure  3-3  shows,  the  majority  of  projects  (68%)  were  new  development,  and 
about  1  in  5  were  maintenance. 


Projects  (%) 

Figure  3-3  Project  Life-Cycle  Phase  Profile 


CMU/SEI-92-TR-24 


13 


Figure  3-4  shows  a  product  size  profile  for  the  projects.  The  projects  are  roughly  evenly 
divided  among  the  size  categories  of  small,  medium  and  large,  with  small  projects  (<100 
KSLOC)  being  the  largest  single  category.  Very  large  projects  were  a  small  percentage  (9%) 
of  the  total  group. 


9% 


29% 


■  Small  (<100  KSLOC) 

□  Medium  (100--299  KSLOC) 
1  Large  (300-999  KSLOC) 

H  Very  Large  (>1000  KSLOC) 


Figure  3-4  Product  Size  Profile 

Figure  3-5  shows  the  peak  project  staffing  profile  for  the  projects.  In  some  cases  this  was 
actual,  and  in  others  it  was  projected,  depending  on  where  the  project  was  in  its  lifecycle. 
Note  that  34%  of  the  projects  included  in  assessments  have  been  been  staffed  by  teams  of 


6%  3% 


38% 


B  No  Response 
□  Small  (0-9) 

E3  Medium  (10-29) 
S  Urge  (30-99) 

E3  Very  Large  (2100) 


Figure  3-5  Peak  Project  Staffing  Profile 
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fewer  than  ten  people.  This  is  consistent  with  our  observation  that  although  the  way  in  which 
process  management  principles  are  implemented  varies  by  project  (and  site)  size,  there  is 
little  disagreement  concerning  the  fundamental  applicability  of  process  management 
principles  and  concepts  independent  of  size. 

3.4  Site  Software  Process  Maturity  Profile 

The  software  process  maturity  level  measures  the  extent  to  which  a  site  has  institutionalized 
continuous  process  improvement  in  its  key  software  processes.  Figure  3-6  displays  the 
profile  of  software  process  maturity  for  the  59  participating  sites  as  reported  by  the  teams 
conducting  the  assessments. 


Figure  3-6  Site  Software  Process  Maturity  Profile 

As  is  apparent  from  the  chart,  none  of  the  participating  software  sites  were  performing  at  the 
Managed  level  (level  4)  or  the  Optimizing  level  (level  5)  at  the  time  their  assessments  were 
conducted. 
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Figure  3-6  is  based  upon  results  from  two  kinds  of  assessments:  SEI-assisted  assessments 
and  self-assessments.  Figure  3-7  shows  the  maturity  profile  broken  out  by  assessment  type. 
The  sites  which  conducted  SEI-assisted  assessments  show  a  more  favorable  profile  in 
comparison  to  sites  which  conducted  self-assessments.  The  most  significant  factor 
accounting  for  these  differences  is  the  selection  criteria  which  were  employed  by  the  SEI  in 
selecting  candidate  organizations  for  SEI-assisted  assessments.  Organizations  receiving 
SEI-assisted  assessments  were  limited  to  those  considered  to  be  doing  advanced  software 
work  and/or  those  organizations  which  were  of  particular  importance  to  the  DoD. 

An  important  related  point  is  that  the  profile  shown  in  Figure  3-6  represents  the  general  state 
of  practice  (of  the  group  of  participating  sites)  from  a  software  process  maturity  perspective 
as  opposed  to  the  leading  edge  of  practice.  Currently,  the  SEI-assisted  assessment  portion 
of  Figure  3-7  is  the  closest  approximation  we  have  to  the  latter. 


I  SEI-Assisted  Assessments  EH  Self-Assessments 


100 


Initial  Repeatable  Defined  Managed  Optimizing 

Figure  3-7  Site  Software  Process  Maturity  Profile  Breakout 
By  Assessment  Type1 


thirteen  sites  encompassing  63  projects  conducted  SEI-assisted  assessments:  46  sites 
encompassing  233  projects  conducted  self-assessments. 
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4  Assessment  Findings  Analysis 


Section  3  focused  on  selected  project  and  site  demographics  and  the  process  maturity 
status  of  the  participating  sites.  This  section  provides  more  visibility  into  the  specific  process 
issues  (i.e.,  findings)  identified  by  the  assessment  teams  during  assessments  of  the  59 
participating  sites  by  analyzing  them  from  the  perspective  of  the  SEI's  Capability  Maturity 
Model  for  Software  VI. 0  (CMM). 

4.1  Rationale  for  Findings  Focus  and  Overall  Approach 

The  primary  objective  of  the  assessment  findings  analysis  is  to  provide  greater  visibility  into 
the  specific  types  of  process-related  issues  identified  during  assessments  of  the 
participating  sites. 

By  definition,  findings  are  the  key  process-related  weaknesses  which  the  site  needs  to  focus 
on  next  as  it  strives  to  institutionalize  continuous  process  improvement  (i.e.,  a  continuous 
advance  to  higher  levels  of  process  maturity). 

A  sample  finding  (extracted  from  the  briefing  conducted  at  the  conclusion  of  an  assessment) 
relating  to  the  area  of  project  estimation  could  be: 

•  Lack  of  wide-spread  use  of  formal  procedures  and  tools  for  estimating  software  size,  cost, 
and  schedule,  and 

•  Limited  and  inconsistent  data  collection  and  dissemination  to  support  estimation 

Figure  4-1  illustrates  how  findings  are  derived  (from  an  information  flow  perspective)  during 
the  course  of  an  SEI  assessment.  This  graphic  shows  how  findings  are  the  result  of  the 
assessment  team's  consideration  of  a  considerable  amount  of  information  about  site 
software  practices  and  procedures  in  the  context  of  the  software  process  maturity  model,  the 
specific  circumstances  of  the  site,  and  the  team's  collective  experience  and  judgement. 

As  a  consequence,  the  findings  constitute  a  relatively  robust  and  stable  focal  point  for 
analysis,  in  contrast  to  project  manager  responses  to  the  maturity  questionnaire  or  the 
results  of  discussions  with  practitioners  alone.  In  addition,  we  have  a  high  level  of 
confidence  in  the  accuracy  and  validity  of  the  findings  because  they  were  formulated  in  the 
context  of  information  about  the  real  issues  confronting  managers  and  practitioners  on  a 
daily  basis,  and  were  developed  in  the  context  of  the  SEI  assessment  method,  a  structured 
diagnostic  method.  For  these  reasons,  we  have  chosen  to  focus  on  the  findings  as  an 
important  source  of  additional  information  on  the  specific  issues  faced  by  software  supplier 
organizations. 
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Project  Managers 
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Figure  4-1  Assessment  Findings  Data  Flow 

The  CMM  was  chosen  as  the  framework  for  the  analysis  of  findings  because  it  constitutes 
the  future  basis  for  the  SEI  assessment  method  (and  other  process-related  SEI  products 
and  services)  and  constitutes  a  relatively  well-defined  and  complete  set  of  key  process 
areas  (KPAs).  Appendix  B  provides  a  brief  overview  of  some  of  the  key  CMM-related 
concepts  and  terms  as  well  as  indications  of  where  additional  CMM  information  can  be 
found. 

The  initial  step  in  the  analysis  of  assessment  findings  was  to  decompose  each  finding  into 
KPAs.  As  used  in  this  analysis,  the  KPAs  can  be  seen  as  a  set  of  coordinates  (or 
dimensions)  in  process  space,  in  the  same  way  that  any  point  in  a  plane  can  be 
characterized  by  knowing  its  coordinates  with  respect  to  a  particular  coordinate  system,  we 
mapped  findings  into  KPA  process  space  using  the  CMM  KPA  coordinate  systems.  For 
example,  the  finding  presented  at  the  beginning  of  this  section  maps  into  the  project 
planning  and  organizational  process  focus  KPAs.  The  motivation  for  performing  this 
classification  was  to  convert  the  findings  into  common  or  neutral  terms  which  would  facilitate 
comparison  and  analysis.  In  addition,  framing  them  in  terms  of  the  CMM  KPA  categories 
provides  a  bridge  from  the  current  set  of  assessment  data  to  the  “next  generation"  of 
assessment  data  resulting  from  the  conduct  of  CMM-based  SEI  assessments. 

4.2  Analysis  Method  and  Examples 

Two  important  aspects  of  an  SEI  assessment  are  the  preparation  and  delivery  of  a  findings 
briefing  to  the  site  senior  management  team  and  participating  personnel,  and  the 
preparation  and  delivery  of  a  comprehensive  final  written  report.  The  text  from  final  findings 


18 


CMU/SEI-92-TR-24 


briefings,  and  sometimes  final  reports,  was  used  as  the  basis  for  this  analysis.  Each  major 
issue  area  identified  during  an  assessment  was  treated  as  one  finding.  In  general,  each 
such  finding  discussed  one  or  more  related  areas  of  concern. 

Each  finding  was  reviewed  and  then  associated  with  appropriate  KPAs.  In  determining  the 
relevant  KPAs  for  a  given  finding,  we  used  the  fundamental  criteria  that  one  or  more  KPAs 
were  relevant  if  their  presence  (or  satisfaction)  would  have  made  it  unlikely  for  an 
assessment  team  to  have  generated  the  finding.  For  example,  the  finding  presented  in 
Section  4.1  would  not  have  been  generated  if  the  project  planning  and  organizational 
process  focus  KPAs  were  fully  in  place  at  the  site.  The  determination  of  relevancy  required 
the  review  and  consideration  of  the  specifications  of  the  KPAs  contained  in  Weber,  Paulk, 
Wise,  and  Withey's  Key  Practices  of  the  Capability  Maturity  Model  [Weber  91  ]. 

There  were  instances  in  which  a  finding  or  parts  of  a  finding  related  to  issues  which  are  not 
within  the  current  scope  of  the  CMM.  In  these  cases,  the  issue  was  assigned  to  the  catch-all 
"Other"  category.  Multiple  instances  of  the  same  KPA  within  a  set  of  site  assessment 
findings  were  counted  as  one  occurrence;  thus,  the  highest  possible  frequency  for  a  given 
KPA  for  a  site  assessment  is  one. 

For  example,  the  following  finding  was  mapped  to  the  software  configuration 
management  KPA: 

Configuration  management  not  consistently  applied 

•  throughout  the  software  development  life  cycle 

•  across  organizations/projects 

•  for  system  maintenance  and  support 

•  to  vendor  supplied  software 

The  following  finding  was  was  determined  to  have  significant  overlap  with  three 
KPAs — project  planning,  project  tracking  and  oversight,  and  other: 

Ineffective  commitment  process 

•  initial  estimates  become  firm  commitments 

•  commitment  made  without  adequate  technical  participation 

•  little  opportunity  to  renegotiate  commitments 

•  over-emphasis  on  achieving  commitments  at  the  expense  of  everything  else 

•  minimal  client  acceptance  of  the  methodology 

•  insufficient  participation  in  reviews 

This  mapping  of  findings  to  KPAs  resulted  in  a  set  of  KPA  weaknesses  for  each  site.  This 
data  was  then  used  to  construct  frequency  distributions  of  KPAs  which  are  the  subject  of  the 
next  section. 

4.3  CMM  Classification  Results 

4.3.1  KPA  Incidence  Distribution  by  Composite  KPA  Maturity  Level 

Figure  4-2  shows  the  distribution  of  KPAs  for  all  59  sites  by  composite  KPA  maturity  level. 
The  individual  KPAs  are  not  shown  in  this  view  of  the  data.  Since  this  chart  is  based  on  the 
total  number  of  KPA  findings  (547  of  them),  it  is  apparent  that  the  bulk  of  the  issues 
considered  most  important  by  the  assessment  teams  are  level  2  and  level  3  issues.  This  is 
consistent  with  our  expectations  based  on  the  maturity  level  profile  presented  in  Figure  3-6, 
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which  showed  93%  of  the  sites  to  be  at  level  1  or  2  and  therefore  facing  level  2  and  3  KPA 
issues.  Figure  4-2  shows  85%  of  the  KPA  findings  to  be  in  the  level  2  or  3  categories.  In 
Section  4.4.1,  we  will  present  additional  information  which  helps  us  better  understand  why 
an  apparently  disproportionate  number  of  findings  are  at  level  3  instead  of  level  2.1 2 


Other 


Level  5  KPA 


Level  4  KPA 


Level  3  KPA 


Level  2  KPA 
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Percent  of  Total  KPA  Findings 
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Figure  4-2  KPA  Incidence  Distribution  by  Composite  KPA  Maturity  Level2 


4.3.2  Individual  KPA  Incidence  Distribution 

Figure  4-3  shows  the  frequency  distribution  of  individual  KPAs  across  all  59  sites  with  the 
KPA  categories  presented  in  maturity  level  order  (level  2  KPAs  are  the  bottom,  then  level  3 
KPAs  and  so  on).  Note  that  this  chart  shows  the  percentage  of  sites  which  had  KPA  findings 
in  the  various  categories  shown. 

The  following  general  observations  can  be  made  from  this  view  of  the  data: 

•  The  five  most  frequently  occurring  KPA  findings  were 
-  product  engineering  (93%) 


1  Figure  3-6  shows  81%  of  the  sites  to  be  at  maturity  level  1  and  1 2%  at  level  2. 

2Note  that  the  bar  patterns  used  in  this  figure  are  primarily  for  helping  to  make  Figure  4-3  more 
readable:  the  patterns  are  carried  over  into  that  figure  from  this  one. 
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-  project  planning  (86%) 

-organization  process  definition  (76%) 

-  project  tracking  and  oversight  (75%) 

-  training  program  (73%) 

•  The  five  least  frequently  occurring  KPA  findings  were 

-  process  change  management  (0%) 

-  defect  prevention  (3%) 

-  subcontract  management  (8%) 

-quality  management  (14%) 

-  peer  reviews  (22%) 

•  In  general,  our  expectation  that  level  4  and  level  5  KPAs  would  have  a  low 
incidence  rate  is  confirmed;  however  the  incidence  of  process  measurement  and 
analysis  (41%)  and  technology  innovation  (24%)  is  relatively  high. 

•  Given  the  high  percentage  of  level  1  sites  (81%),  subcontract  management  (a  level 
2  KPA)  does  not  appear  to  be  a  significant  problem  area  for  these  sites  (with  an 
incidence  of  only  8%). 

•  Similarly,  the  incidence  level  of  peer  reviews  (22%)  is  lower  than  would  be 
expected  given  the  number  of  sites  assessed  to  be  at  levels  1  and  2. 

•  Just  over  half  of  the  sites  were  considered  (by  the  assessment  team)  to  be  facing 
important  issues  which  fall  outside  the  current  scope  of  the  CMM  ("Other"  category 

-  56%).  Examples  of  such  issues  include  organizational  structure  and  failure  of 
previous  management  initiatives. 
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Other 


Technology  Innovation 
Defect  Prevention 
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Figure  4-3  Individual  KPA  Distribution 
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4.4  Understanding  and  Interpreting  the  KPA  Distributions 


There  are  a  number  of  factors  to  consider  when  interpreting  and  understanding  the  KPA 
distributions  presented  above.  In  this  section  we  will  explain  how  limiting  the  number  of 
assessment  findings,  along  with  the  KPA  structure  of  the  CMM,  impacts  the  KPA 
distributions  presented  previously.  Then  we  will  present  rationale  for  some  of  the 
observations  made  about  the  individual  KPA  distributions.  Finally,  we  will  introduce 
additional  considerations  believed  to  have  an  impact  as  well  as  identifying  other  factors 
which  might  be  significant. 


4.4.1  Effect  of  Number  of  Findings/KPAs  Identified  on  Assessments 

Assessment  teams  are  trained  to  limit  the  number  of  findings  to  a  manageable  level  (5-9  is 
the  range  suggested  by  the  SEI)  and  to  prioritize  them  as  per  the  precedence  relation 
implicit  in  the  maturity  model.  Sites  are  then  advised  to  focus  their  improvement  efforts  on 
the  highest  priority  issues  first.  We  would  expect  these  issues  for  the  most  part  to  map  to 
findings  related  to  KPAs  at  the  maturity  level  one  above  the  sites’  current  rating. 

One  of  the  implications  of  this  guidance  is  that  a  site  may  have  findings  which  map  into 
process  areas  which  are  beyond  what  the  site  needs  to  focus  on  to  advance  to  the  next 
higher  maturity  level.  The  ’’closer”  the  site  currently  is  to  the  next  higher  maturity  level,  the 
higher  the  proportion  of  findings  which  address  issues  related  to  KPAs  at  maturity  levels  two 
or  more  above  the  site’s  current  rating. 

For  example,  we  would  generally  expect  that,  in  the  frequency  distribution  of  KPAs  for 
maturity  level  1  sites,  there  will  primarily  be  instances  of  maturity  level  2  process  issues,  plus 
some  level  of  incidence  of  level  3  process  issues.  Similarly,  for  the  frequency  distribution  of 
process  issues  for  maturity  level  2  sites,  there  will  primarily  be  instances  of  maturity  level  3 
process  issues,  plus  some  level  of  incidence  of  level  4  process  issues. 

Figures  4-4  and  4-5  show  the  breakdown  of  findings  and  KPAs  from  an  occurrence 
perspective.  That  is,  Figure  4-4  shows  the  number  of  assessments  that  had  a  specific 
number  of  findings.  Similarly,  Figure  4-5  shows  the  number  of  assessments  that,  as  per  our 
mapping  from  findings  to  KPAs,  had  a  specific  number  of  KPAs  indentified  as  issue  areas. 

In  general,  the  results  in  Figure  4-4  are  generally  consistent  with  our  expectations  that  most 
assessments  would  yield  between  five  and  nine  findings  —  74%  of  the  assessments  fit  into 
this  range.  The  average  number  of  findings  per  assessment  by  various  groupings  is  as 
follows: 

•  across  all  sites  (59)  =  7.6 

•  level  1  sites  (48)  =  7.8 

•  level  2  sites  (7)  =  7.0 

•  level  3  sites  (4)  =  6.5 
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Figure  4-4  Number  of  Findings  Identified  on  SEI  Assessments 
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Figure  4-5  Number  of  KPAs  Identified  on  SEI  Assessments 
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Since  the  CMM  KPA  framework  was  not  explicitly  utilized  in  any  of  the  59  assessments  we 
are  considering,  there  are  no  a  priori  expectations  about  the  number  of  KPAs  which  would 
surface  as  issues  during  an  assessment.  Based  on  the  assessment  finding-to-KPA  space 
mapping,  the  average  number  of  KPAs  per  assessment  by  various  groupings  is  as  follows: 

•  Across  all  sites  (59)  =  9.3 

•  Level  1  sites  (48)  =  9.6 

•  Level  2  sites  (7)  =  7.7 

•  Level  3  sites  (4)  =  8.0 

The  key  point  of  this  analysis,  which  helps  explain  the  distribution  of  KPAs  shown  in  Figures 
4-2  and  4-3,  is  that  we  should  expect  level  1  sites  to  have,  on  average,  3  or  4  KPAs  cited  as 
issues  which  are  beyond  level  2  (since  there  are  six  level  2  KPAs  —  refer  to  Figure  B-1).  In 
addition,  since  we  expect  some  level  1  sites  to  be  closer  to  level  2  than  others,  we  expect 
that  some  percentage  of  the  level  1  sites  will  have  more  than  3  or  4  KPAs  in  the  level  3  or 
above  category. 

We  expect  the  impact  of  spilling  over  into  the  next  higher  maturity  level  to  be  less 
pronounced  for  sites  rated  at  maturity  level  2  since  there  are  seven  KPAs  at  level  3  and  the 
level  2  sites  averaged  7.7  KPAs  per  assessment. 

4.4.2.  Other  Factors 

There  have  been  instances  where  sites  are  rated  at  a  particular  maturity  level  in  spite  of  the 
presence  of  findings  at  a  lower  maturity  level.  This  is  typically  done  when  the  assessment 
team  feels  the  site  is  borderline  and  decides  to  give  it  the  benefit  of  the  doubt.  For  this 
reason,  we  expect  that  the  frequency  distributions  will  also  show  an  incidence  of  process 
areas  at  a  level  lower  than  that  of  the  site's  overall  maturity  rating. 

We  also  expect  to  see  some  anomalous  variations  deriving  from  the  complexity  and  difficulty 
of  an  assessment  plus  the  inherent  variation  deriving  from  its  human-intensive  nature  (e.g., 
the  occurrence  of  level  4  or  5  process  areas  at  level  1  sites,  or  some  occurrence  of  level  5 
process  areas  for  level  2  sites). 

Other  factors  that  might  be  significant  to  the  KPA  profiles  include: 

•  Differences  between  the  original  maturity  model  (Figure  2-1)  and  the  CMM 
(Figure  B-1) 

•  Current  KPA  partitioning  and/or  their  maturity  level  associations 

•  Our  mappings  from  findings  space  into  KPA  space 

•  Definition  of  the  SEI  assessment  method 

•  Extent  to  which  assessment  teams  followed  SEI  guidance 

•  Ability  of  assessment  teams  to  identify  or  differentiate  findings  that  are  consistent 
with  the  site's  maturity  level 

•  Assessment  team  bias  for  technological  solutions 

The  extent  to  which  these  (or  other  factors)  are  significant  is  not  well  understood  at  this  time. 
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4.5  Findings  Analysis  Conclusions 


With  a  few  minor  exceptions,  our  qualitative  understanding  of  the  findings  distribution  in  KPA 
space  is  good.  It  is,  in  general  terms,  consistent  with  the  site  maturity  level  profile. 

On  average,  site  assessments  identified  7.6  findings  and  9.3  KPA  findings.  Because  of  this 
the  findings  usually  span  at  least  two  maturity  levels. 

The  five  most  frequently  occurring  findings  areas  are  product  engineering,  project  planning, 
organization  process  definition,  project  tracking  and  oversight,  and  training  programs.  The 
five  least  frequently  occurring  findings  areas  are  process  change  management,  defect 
prevention,  subcontract  management,  quality  management,  and  peer  reviews. 
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Appendix  A  Participating  Organizations 


The  27  organizations  shown  in  Table  A  collectively  provided  the  SEI  with  results  from  SEI 
assessments  conducted  at  one  or  more  of  their  software  sites.  Organizational  ranking 
(within  the  top  100  prime  Defense  Department  contractors  for  fiscal  year  1990  based  on  net 
contract  value)  and  total  contract  value  data  are  taken  from  Carroll  Publishing  Company's 
DEFENSE  Industry  Services  [Carroll  91]. 
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Organization  Name 

Top  100 
Ranking 

Total  Contract  Value 
(K$s) 

ALCOA 

NA 

NA 

The  Boeing  Company 

11 

2267 

Computer  Sciences  Corporation 

53 

319 

Loral  Corporation  (Ford  Aerospace) 

31 

618 

General  Dynamics 

2 

6306 

General  Electric  Company 

3 

5589 

GTE  Corporation 

17 

1294 

Harris  Corporation 

71 

188 

Honeywell  Inc. 

15 

1388 

General  Motors  Corporation  (Hughes  Aircraft  Company) 

4 

4107 

IBM  Corporation 

18 

1286 

Jet  Propulsion  Laboratory 

NA 

NA 

LTV  Corporation 

20 

1183 

McDonnell  Douglas  Corporation 

1 

8211 

Philips  (Magnavox  Electronic  Systems  Company) 

56 

297 

Medtronic 

NA 

NA 

Motorola  Inc. 

45 

403 

Northrop  Corporation 

26 

746 

Pacific  Bell 

NA 

NA 

Software  Productivity  Consortium 

NA 

NA 

Texas  Instruments  Inc. 

29 

704 

TRW  Inc. 

22 

1087 

Unisys  Corporation 

16 

1376 

US  Air  Force 

NA 

NA 

US  Army 

NA 

NA 

US  Navy 

NA 

NA 

Westinghouse  Electric  Corporation 

12 

2243 

Total  Contract  Value 

NA 

39612 

Table  A  Participating  Organizations 


30 


CMU/SEI-92-TR-24 


Appendix  B  About  the  CMM 


As  noted  in  Section  2,  the  SEI  has  released  an  elaboration  and  refinement  of  the  SEI 
process  maturity  model  ([Paulk  91],  [Weber  91]),  referred  to  as  the  SEI  Capability  Maturity 
Model  for  Software  VI. 0  (CMM).  In  the  CMM,  maturity  levels  2  through  5  are  characterized 
by  a  set  of  key  process  areas  (KPAs)  as  shown  in  Figure  B-1.  In  order  for  a  site  to  be 
considered  to  be  performing  at  a  given  level  of  maturity,  it  must  be  determined  that  the  site 
has  shown  a  specific  level  of  competency  in  each  of  the  associated  key  process  areas.  For 
example,  if  a  site  was  determined  to  be  deficient  in  the  area  of  software  project  planning  (as 
specified  in  Weber,  Paulk,  Wise,  and  Witney  [Weber  91]),  then  it  would  be  considered  to  be 
performing  at  the  initial  level  of  process  maturity. 


Maturity  Level 

Characteristic 

Key  Process  Areas 

5 

Optimizing 

Continuous  process 
capability 
improvement 

Process  change  management 

Technology  Innovation 

Defect  prevention 

4 

Managed 

Product  quality 
planning  and  tracking 
of  measured  software 
process 

Quality  management 

Process  measurement  and  analysis 

3 

Defined 

Software  processes 
defined  and 
institutionalized 
to  provide  product 
quality  control 

Peer  reviews 

Intergroup  coordination 

Software  product  engineering 

Integrated  software  management 

Training  program 

Organization  process  definition 

Organization  process  focus 

2 

Repeatable 

Management  oversight 
and  tracking  of  project; 
stable  planning  and 
product  baselines 

Software  configuration  management 

Software  quality  assurance 

Software  subcontract  management 

Software  project  tracking  and  oversight 
Software  project  planning 

Requirements  management 

1 

Initial 

Figure  B-1  Key  Process  Areas  of  the  Capability  Maturity  Model 

Figure  B-2,  taken  from  Paulk,  Curtis,  Chrissis,  et  al’s  Capability  Maturity  Model  for  Software 
[Paulk  91],  provides  an  example  of  how  the  concepts  of  maturity  level,  key  process  area, 
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and  goal  relate  to  one  another  in  the  CMM,  using  the  KPA  of  software  project  planning  as 
the  context. 


f  il  Maturity  Level: 


Level  2,  Repeatable 

- ^ - 


n 


indicates 


contains 


Goat: 

A  plan  is  developed  that 
appropriately  and  realistically 
covers  the  software  activities 
and  commitments. 


Estimates  for  the  size  of 
software  products  are 
derived  according  to  a 
documented  procedure. 


C Implementation  or 
Institutionalization 
Activity: 

Implement 


uni 

Do  you  use  a  documented  prodedure  to 
estimate  software  size  (e.g.,  lines  of  code, 
function  points,  etc.)? 


Figure  B-2  CMM  Structure  Example  At  Level  2 

Paulk,  Curtis,  and  Chrissis  provide  an  introduction  to  the  Capability  Maturity  Model  for 
Software  [Paulk  91],  and  Weber,  Paulk,  Wise,  and  Withey  provide  an  in-depth  description  of 
the  goals  and  associated  key  practices  for  each  KPA  [Weber  91]. 
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